fix: max tokens #820
fix: max tokens #820
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
| super(); | ||
| this.messages = init?.messages ?? []; | ||
| this.messagesBefore = this.messages.length; | ||
| this.summaryPrompt = init?.summaryPrompt ?? defaultSummaryPrompt; | ||
| this.llm = init?.llm ?? new OpenAI(); | ||
| if (!this.llm.metadata.maxTokens) { | ||
| if (!this.llm.metadata.maxTokens || !init?.maxTokens) { |
There was a problem hiding this comment.
Should we use Infinity if it's undefined?
There was a problem hiding this comment.
the code below won't work if maxTokens is not specified, so I think it's ok, to throw this error, it's basically a configuration error.
this class also needs a tokenizer to work correctly (and for ollama it could be any LLM with any tokenizer).
So I think we should set tokenizer to globalsHelper.defaultTokenizer.encode only if the LLM is OpenAI and throw an error if tokenizer is not explicitly specified otherwise
himself65
force-pushed
the
himself65/20240508/max-token
branch
from
e61e843
to
f1906bf
Compare
Fixes: https://discord.com/channels/1059199217496772688/1133167189860565033/1237394506274439268